Software Vault: The Gold Collection

home *** CD-ROM | disk | FTP | other *** search

/ Software Vault: The Gold Collection / Software Vault - The Gold Collection (American Databankers) (1993).ISO / cdr52 / bcsmem.zip / MEMORY.DOC < prev next >

Wrap

Text File | 1993-03-27 | 53KB | 1,134 lines

(10U Clipper Memory Management by Roger Donnay A. Linking Techiques for Memory Management 1. Dynamic Overlaying of Clipper Code 2. Dynamic Overlaying of C/ASM Code 3. Using EMS or UMB Ram for Overlays 4. Static Overlays 5. Reloadable Overlays 6. Overlaying Library Modules 7. Speed Optimization B. Programming Techniques for Memory Management 1. Symbol Management 2. Memory effects of Database usage ╔════════════════════════════════════════════╗ ║ LINKING TECHNIQUES FOR MEMORY MANAGEMENT ║ ╚════════════════════════════════════════════╝ The Clipper compiler has the responsiblity of converting source code to "machine-readable" code. This .OBJect code can not be "executed" by the machine however until it is "linked" to other .OBJect code segments in the Clipper libraries. This is the responsibility of the linker. Linking or "fixing up" references made by your code to external variables and functions is necessary to insure that your compiled program will address or call other programs, including functions in the Clipper libraries and DOS functions. A linker's basic job is to combine objects which have been compiled separately. Objects contain symbols of three types: a. PUBLIC symbols - those symbols which are callable from other objects b. STATIC symbols - those symbols which are callable only from within the same object c. UNDEFINED symbols - symbols which exist in other objects and are referenced or used by this object (also referred to as EXTERNAL symbols). The Role of Linkers in developing large applications has taken a new turn in the past few years. In addition to providing segment-linking, they are now also expected to provide the best "memory image" possible. This means that the linker must be sophisticated in it's management of code segments during the running of an application. The physical size of a Clipper application is no longer representative of it's memory image, because the linker "overlays" the code segments to use the smallest possible amount of memory. Some type of code (such as Clipper P-CODE) is easy to overlay while other types (ASM) may be more difficult. Applications that are almost all Clipper-compiled code will usually produce the best memory image regardless of the physical size of the .EXE. The subjects of this seminar are the two most commonly-used linkers available on the market today for linking Clipper 5.0 applications. They are: RTLINK - A Clipper-5.0 specific linker developed by Pocketsoft, Inc. This is the linker that is bundled with Clipper 5.0. BLINKER - A Third-Party Clipper-specific linker published by Blink, Inc. for Summer 87 and 5.0 applications. ╔══════════════════════════════════════╗ ║ DYNAMIC Overlaying of Clipper Code ║ ╚══════════════════════════════════════╝ Very large Clipper applications can consist of more code than can be loaded into the available amount of DOS memory and may not be able to execute due to insufficient memory. Clipper 5.0 has a "Dynamic Overlay Manager" actually built into the Clipper libraries that helps prevent this problem by loading the code into memory only as it is needed by the application that is running. The linker helps prepare the application to use this capability of 5.0 by splitting the Clipper-compiled application code into fixed-size "pages" that share a pre- allocated area of memory. These pages of code are then written to a seperate .OVL file or to the end of the .EXE file as determined by your linker options. When a procedure or function is called, the "dynamic-overlay manager" first checks to see if it is in memory. If it is, then it jumps to the called function. If it is not in memory, it loads the function from disk into any "free" location available in the overlay pool. If no free space can be found in the overlay pool, the overlay manager discards pages that have not been called recently to free space to load the called procedure. The larger the overlay pool, the less often the overlay manager needs to load from disk and the faster your program will run. The size of the dynamic overlay pool needed for your application is automatically determined by Clipper. The great thing about Clipper 5.0's dynamic overlaying system is that you, the programmer, don't need to do anything to create overlays. It is all done automatically by the linker. Only Clipper-compiled modules, not C/ASM modules, are overlayed "dynamically", although since much of the Clipper libraries has been compiled in Clipper, even large portions of the Clipper libraries will be dynamically overlayed. Application libraries and third-party libraries written in Clipper will also be automatically overlayed by RTLINK and BLINKER. In many cases it is more beneficial to write functions in Clipper (using LOCAL memvars) than in C or ASM because it will not increase the executable memory model. Even Clipper code that has been linked into to Pre-link libraries will be dynamically overlayed. INTERNAL/EXTERNAL Overlays --------------------------- INTERNAL overlays are dynamic pages which are written to the end of the .EXEcutable program file. During the running of the application, the dynamic overlay manager reads the disk and loads dynamic pages from this file. This is the default overlay method of RTLINK and BLINKER and it is often advantageous to use internal overlays to reduce the number of file handles being opened by your application. In some cases however, you may find that using multiple overlay files will increase the speed performance of your application because extracting code from several small files which are all opened at the same time is often faster that extracting code from one large file. EXTERNAL overlays are sets of dynamic pages which are written to a seperately designated file more commonly referred to as an overlay file. The positional command /DYNAMIC[:<ovlfile>] or the freeformat command DYNAMIC INTO <ovlfile> will tell RTLINK to write the dynamic pages to a file named <ovlfile>. You may wish to create a seperate overlay file when your application .EXE is too large to fit on a distribution disk. Examples: Rtlink> DYNAMIC INTO myprog.ovl Blinker> SECTION INTO myprog.ovl Rtlink> DYNAMIC INTO myprogs.ovl MYPROG1, MYPROG2 DYNAMIC INTO myfuncs.ovl MYFUNC1, MYFUNC2 Blinker> SECTION INTO myprogs.ovl MYPROG1, MYPROG2 SECTION INTO myfuncs.ovl MYFUNC1, MYFUNC2 ╔══════════════════════════════════════╗ ║ DYNAMIC Overlaying of C / ASM Code ║ ╚══════════════════════════════════════╝ BLINKER also provides the ability to dynamically overlay code written in C or Assembly. This is the most common code found in many third-party libraries. RTLINK also provides for overlaying of C/ASM but it is more complex and requires understanding more about the code being overlayed. See STATIC OVERLAYS and RELOADABLE OVERLAYS for more information on overlaying C/ASM code with RTLINK. In the previous chapter, we discussed that application code compiled with the Clipper-compiler is dynamically overlayed using the overlay manager built inside the Clipper libraries. This Clipper-specific internal overlay manager is incapable of overlaying C/ASM code, so it is the responsibility of the linker to provide this capability. Each linker has it's own methods for overlaying this kind of code. BLINKER uses an overlay pool that is allocated at the start of the application. The size of this pool is determined at link-time. The larger the overlay pool, the faster the application will run because overlays will not be reloaded into memory from disk so often. The dynamic-overlay capabilities of BLINKER require that the objects being overlayed are "well-behaved". After much experimentation with our own libraries and many third-party libraries, it is now more clear what constitutes well-behaved modules: a. Well-behaved code uses the Clipper EXTEND interface, registers or the stack for parameter passing and does not use undocumented features of Clipper. b. All Clipper-compiled .OBJects are dynamically- overlayable. Modules which CANNOT be dynamically-overlaid are: a. Routines which handle interrupts. b. Modules in which the DATA area is changed during runtime rather than allocating memory in the root area. Dynamic overlaying of third-party libraries can be an aggravating learning experience if you don't get support from the third party vendor in trying to accomplish this task. Most of the third-party vendors now break their libraries into separate overlayable and non-overlayable .LIB files to make the task much easier. Once you have determined which .OBJ files and/or .LIB files are acceptable for overlaying, the task of telling the linker to do this is quite simple. A BLINKER link-script will look similar to an RTLINK script except there is always only one (1) overlay area because all code uses the same overlay "pool". Example of a BLINKER script for overlaying C/ASM: # Allocate 50k to the overlay pool BLINKER OVERLAY OPSIZE 50 BEGINAREA # Clipper-compiled application code FILE myprog1.obj, myprog2.obj # C/ASM files FILE ccode.obj, asmcode.obj # C/ASM libraries LIB ccode.lib, asmcode.lib ENDAREA ╔══════════════════════════════════════════╗ ║ Using EXPANDED or UMB RAM for OVERLAYS ║ ╚══════════════════════════════════════════╝ BLINKER provides a several very useful feature for increasing the amount of available memory for Clipper applications, i.e., the use of Expanded (EMS) or Upper Memory Blocks (UMB). -- EMS -- Many computers have EMS memory. EMS memory managers such as QEMM, 386MAX and DOS 5.0 create a memory area above conventional memory referred to as the "page frame". This 64k block of memory is mapped to an area of ram that can execute 8086 code. Blinker takes advantage of this feature by allocating the 64k EMS page frame rather than conventional DOS memory for their runtime overlay pool. This frees an additional 64k of memory for use by Clipper because the overlays are loaded into the page frame rather than conventional ram. If no EMS is present when the application starts, then the overlay pool will be established in normal (conventional) ram. To enable this feature, use the Blinker command: BLINKER OVERLAY PAGEFRAME ON -- UMB -- Expanded memory managers will also create an area between the 640k and 1 meg memory address called UMB (upper memory blocks). This is the area where you would normally load your memory- resident programs such as DOS, network-drivers, etc. If your enviroment gives you about 40k - 60k of contiguous memory available in the UMB area, Blinker will load the overlay pool into the UMB area rather than conventional area, thus freeing more memory for your Clipper application. To enable this feature, use the Blinker command: BLINKER OVERLAY UMB ON ╔═══════════════════╗ ║ STATIC Overlays ║ ╚═══════════════════╝ STATIC overlays are supported by RTLINK only. They provide a method of overlaying C and ASM code which cannot be overlayed by the dynamic-overlay manager. C and ASM code is not as easily relocatable as Clipper compiled PCODE (pseudo-code) therefore it cannot be overlayed "dynamically". A function which is dynamically loaded and unloaded from a pool is usually found in different places in memory at different times during the running of the program. When a C or assembly routine is called by an application linked with RTLINK it must be loaded in memory at the same address every time. This is accomplished by designating overlay AREAS where many different modules will occupy the same memory space, but only one-at-a-time. Designing Static overlays requires a more thorough understanding of the structure of your program. Static overlay segments usually look like this in your link file: # area 1 BEGINAREA SECTION FILE A SECTION FILE B SECTION FILE C ENDAREA # area 2 BEGINAREA SECTION FILE D SECTION FILE E SECTION FILE F ENDAREA You must be sure that you properly place your objects in the overlay segments to prevent computer "lock-up" or "crashing" at runtime. In the above example, any procedure or function in FILE A may call any procedure or function in FILE D, E or F because they are in different areas, however if a function in FILE A calls a procedure or function in FILE B, then your program will crash as soon as the program returns to FILE A because FILE A was removed from memory to load FILE B. Overlay management with "static overlays" is the most time- consuming because it requires extensive analysis of the structure of the program to insure that the modules are placed properly in the overlay areas. ╔═══════════════════════╗ ║ RELOADABLE Overlays ║ ╚═══════════════════════╝ The following overlaying technique may be considered controversial because it is an "unsupported" and "undocumented" feature of the RTLINK linker supplied with Clipper-5.0. I feel however, that it is important to cover this subject because without an understanding of how to use this feature it may be impossible to develop certain types of Clipper applications. dCLIP is an example of a Clipper application that would not have been possible if I could not use "reloadable" overlays. Please be aware that this subject material is not supported by Nantucket and is provided as information to be used at your own risk. I have been RELOADING objects from the Summer 87 Clipper libraries for years, so when I received my copy of Clipper-5.0, the first thing I attempted to do was to overlay some of the larger C/ASM modules in the Clipper libraries using the RELOAD command in my linker script file. The Nantucket development team had the foresight to insure that their C and ASM compiler assigned a UNIQUE name to each Clipper C/ASM object in the Clipper libraries, thus allowing, you, the Clipper programmer to use both the RELOAD command and MODULE command in your .LNK script files. So what are "Reloadable Overlays"? These are a form of STATIC overlay in which segments of code are placed into overlay sections that occupy the same memory space at runtime, however the "calling module" is automatically "reloaded" into memory when returning from the "called module". Reloadable overlay segments usually look like this in your link file: RELOAD FAR 200 # area 1 BEGINAREA SECTION FILE A SECTION FILE B SECTION FILE C ENDAREA # area 2 BEGINAREA SECTION FILE D SECTION FILE E SECTION FILE F ENDAREA Not all Clipper applications are going to need to use these reloadable static overlays, but if your application relies heavily on third-party libraries written in C/ASM, then it is recommended that you learn this technique. The main advantage of "reloadable overlays" over conventional "static overlays" is that you are never in danger of "lockup" in the event that a procedure or function in FILE A calls a function or procedure in FILE B because FILE A will be reloaded into memory on return from FILE B. Make sure when you use the RELOAD command that you always use it as follows: RELOAD FAR <stack size> where the <stack size> is the amount of memory in hex bytes to use for saving addresses. My experience is that most applications will run just fine with a stack size of 200. If your application does recursive nesting, however, you may need to increase the stack size. Recursive nesting is described as follows: Procedure A calls Procedure B which calls Procedure A which calls Procedure B, etc, etc, etc. Each time a procedure is called, its return address is "pushed" onto the stack, and each time you return to the calling procedure the address is "popped" from the stack. In the above example, if this recursive condition were allowed to continue for a large number of iterations, the stack would be overrun and the program may crash. Although your application will not crash using RELOAD, it may slow down a bit if overlay segments are not structured properly, because each time an overlay is reloaded the application must go to disk. In the above example if a function in FILE A repetitively calls a function in FILE B then your application will be very "disk intensive" and will run slowly. If a function in FILE A repetitively calls a function in FILE D there will be no slowing at all because both modules will remain in memory. Overlay management of C/ASM code with "reloadable overlays" is less time-consuming and more reliable than simple conventional "static overlays" and provides an additional advantage of allowing more modules to be overlayed, thereby saving additional memory usage. ╔════════════════════════════╗ ║ OVERLAYING LIBRARY MODULES ║ ╚════════════════════════════╝ The MODULE command provides the important capability of overlaying the larger modules of CLIPPER.LIB or any other third-party library. With previous versions of these linkers, the overlaying of libraries was restricted to ALL modules in a library or NO modules in a library. This is ok for libraries which were designed for overlaying, but in most cases it is simply not practical to overlay an entire library due to speed performance problems. There are many modules in CLIPPER.LIB which can be overlayed without crashing the program, yet runtime performance can be disastrous. Determining the proper "set" of modules and the optimum "overlay pool size" is simply a matter of trial and error. For example, overlaying the MACRO module in one application may have very little affect on speed performance because of the design of the program and it's probable negligible On the contrary, it was completely unacceptable to overlay MACRO in the dCLIP engine because dCLIP relies heavily on the MACRO compiler for many of it's interpretive operations. The MODULE command is very useful in that it allows your linker to place "modules" from libraries into overlays. You can organize your projects by placing all your C/ASM objects in libraries with a library manager such as Microsoft's LIB.EXE then by using the SECTION MODULE <module> command in your link file you can place any module into any overlay area. Rtlink: LIB mylib BEGINAREA SECTION MODULE myfileA,myfileB SECTION MODULE myfileC SECTION MODULE myfileD ENDAREA The MODULE <module> command in RTLINK requires that the <module> exists in one of the declared libraries and that the name is "unique" to that module otherwise the command will be ignored and the module will be linked into the root memory area. This is not a problem with the Clipper libraries because the modules are given unique names. BLINKER supports the MODULE command differently and, in my opinion, better than RTLINK. Blinker allows you to define a specific module from a specific library to be linked into the application, thus you don't need to worry about the module being unique to one library. For example, if you have two libraries, both containing an ERRORSYS module, you can decide which ERRORSYS you want linked into the program. Blinker: LIB mylib BEGINAREA LIB grump MODULE errorsys FROM grump MODULE myfileA,myfileB MODULE myfileC MODULE myfileD ENDAREA DETERMINING THE MODULE NAMES IN A LIBRARY ------------------------------------------ Every ".OBJect" file placed into a library is also assigned a "module" name. This module name is stored in the "COMENT" record of the object in the library. The "module" name of an object is assigned by the compiler of the object and is usually given the name of the original source code. The Clipper compiler assigns the same name as the .OBJ file, however many compilers will assign the name of the SOURCE file as the module name including drive letters and directories. For example, let's say we want to place the Funcky FINDATTR() function into an overlay. First, we must figure out what "module" name was assigned to the FINDATTR() function. There are library manager utility programs which can help you figure this out, but I'm going to show you how to do it simply with your linker. During VERBOSE linking the "module" name of each object being linked into the program is displayed on the screen as follows: FUNCKY50.LIB(C_MAXCHO) <- Clipper code FUNCKY50.LIB(C_PUTKEY) <- Clipper code FUNCKY15.LIB(findattr.C) <- "C" code FUNCKY15.LIB(finddate.C) <- "C" code FUNCKY15.LIB(findfirs.C) <- "C" code FUNCKY15.LIB(chrfound.ASM) <- "Assembly" code FUNCKY15.LIB(strcente.ASM) <- "Assembly" code You must make sure that you insert the command VERBOSE into your link script file to insure that this information is displayed. Next, when you perform the link, make sure route the dislay output to a file so you can have the information in text file format: RTLINK @MYPROG > MYPROG.TXT MYPROG.TXT will contain the complete list of modules linked into your program. Now, we must take this information and use it to develop an overlay strategy. The name of each module linked into the program is shown (in parenthesis) and must be referred to "exactly" as it is spelled. For example, to place the function FINDATTR() into an overlay, we must first determine the "module" name which would contain the FINDATTR() function. Fortunately, the FUNCKY15 library uses the same prefix name as the actual function, so in perusing the list of linked objects in MYPROG.TXT we find a module named "findattr.c". To place this module in an overlay section, simply use the command: SECTION MODULE findattr.c BLINKER's MODULE command is more useful than RTLINK's because it searches both the THEADR record and the COMENT record for the name of each module. This means that you don't need to reference the MODULE name as precisely with these linkers as with RTLINK. For example, if the module name in a library is FINDATTR.C, BLINKER will overlay it by with the following command: MODULE FINDATTR, because it will determine that what you are trying to overlay is FINDATTR.OBJ. OVERLAYING THE CLIPPER LIBRARIES --------------------------------- Overlaying the Clipper libraries is accomplished quite differently with each linker, therefore I have included examples for each linker. ---------- RTLINK ---------- RTLINK uses Static reloadable overlays with and the MODULE command to overlay much of the Clipper libraries and thereby reduce memory usage in your Clipper-5.0 applications. The below link script example creates an overlay area for overlaying the larger modules in the Clipper libraries which are not likely to call each other recursively, therefore you will probably notice very little difference in speed performance yet you will get up to 50K more memory overhead depending on how much of the Clipper libraries your application uses. You may get a "warning" message during link time if your application does not call one of the modules referenced in an overlay area. In the event this happens, simply remove that module from the link file. For example, if you are not using MEMOEDIT() in your application remove the line: SECTION MODULE 50\MEMOEDIT FI <my files> LIB <my libs> OUTPUT <my .EXE> RELOAD FAR 200 BEGINAREA SECTION MODULE 50\MEMOEDIT, 50\MEMOTRAN, 50\MEMOREAD MODULE 50\MEMOWRIT, 50\MEMOLINE, 50\MLCOUNT MODULE 50\MLPOS SECTION MODULE 50\IS, 50\EXAMPLEC, 50\HARDCR, 50\PAD MODULE 50\PADL, 50\PADC, 50\PADR, 50\ASCAN MODULE 50\ASORT, 50\DIRECTRY, 50\AEVAL, 50\ACOPY MODULE 50\ADEL, 50\AINS, 50\COPYFILE, 50\TYPEFILE MODULE 50\SCROLL, 50\GETE, 50\DISKSPAC SECTION MODULE 50\SDF1, 50\SDF0, 50\SDFDYN, 50\DLM1 MODULE 50\DLM0, 50\DELIMDYN, 50\DBCREATE MODULE 50\DBJUNCT, 50\DBSTRUCT, 50\ACHOICE SECTION MODULE 50\NET, 50\ACCEPT, 50\DATE, 50\OSDATE SECTION MODULE 50\FGET, 50\OLDBOX, 50\OLDCLEAR, 50\RUN MODULE 50\SEND, 50\JOINLIST, 50\SORTOF, 50\SORTBLOC ENDAREA BEGINAREA SECTION MODULE 50\TBROWSE0 SECTION MODULE 50\TBROWSE1 SECTION MODULE 50\EXAMPLEA, 50\PHILES, 50\DBF0, 50\DTX0 MODULE 50\INITEXIT, 50\TXOPEN, 50\BOX, 50\STRTRAN ENDAREA When placing modules into overlays, you must use extreme care to not overlay functions which are called by "interrupts" or may be called "recursively". Code which is called by interrupts is called by a direct vector rather than an overlay vector this can cause computer lockup if the code segment is not currently in memory. Code which is called recursively would be something like special string handling routines that might be called in a do..while loop. If this function were placed in an overlay, you may experience considerable slowing and/or disk access. Many third-party library developers now distribute their product in two libraries: a "resident" library and an "overlayable" library. Most of the work has already been done for you in determining which modules can be overlayed. If the module is part of the "resident" library, don't make any attempt to try to overlay it or you will experience runtime problems. Also, don't make an attempt to overlay "Clipper code" because this code is automatically overlayed into "dynamic-pages". ----------- BLINKER ----------- Overlaying the Clipper libraries with Blinker is simpler than Rtlink because only one overlay area needs to be defined for all overlayable modules, files and libraries. Like Rtlink, Blinker will automatically overlay any code in any library that is compiled with the Clipper compiler. BLINKER OVERLAY OPSIZE 50 BLINKER PROCEDURE DEPTH 50 FI <my root files> LIB <my root libs> OUTPUT <my .EXE> BEGINAREA FILE <my overlayable files> ALLOCATE <my overlayable libraries> # Modules from CLIPPER.LIB MOD accept,acopy,adel,aeval,ains,appinit,atail,box,cmem,date MOD dbcmd2,dbcmd3,dbcmd4,dbcmd5,dbcreate,dbf0,dbfdyn,dbjunct MOD dbstruct,dtx0,dtx1,dtxdyn,dynina,errsys0,errsys1,fget MOD getenv,gets0,gets1,gets2,gx,joinlist,lupdate,memory MOD mrelease,msave,net,oldbox,oldclear,philes,run,saverest MOD sdf0,sdf1,seq,setcurs,sortbloc,startsym,tb,txopen,vall MOD vblock,vdb,vdbg,version,vmacro,vnone,vops,vpict,vterm MOD workarea,xmacro ALLOCATE extend ENDAREA LIB clipper, terminal ,dbfntx ╔══════════════════════╗ ║ SPEED OPTIMIZATION ║ ╚══════════════════════╝ Most features of the new generation of linkers are designed for memory optimization. Unfortunately, improving memory performance "almost always" works against speed performance when the job is being handled by the linker. Overlaying of code requires that code segments be swapped in and out of memory and that "overlay vectors" rather than "direct vectors" are used to access the code. This increased overhead will slow down the application and often-times create unacceptable runtime performance. Simply overlaying an entire library is not always the best choice when speed performance is critical. For example, a function that skips a record pointer through a database must be called once for each record. If this function is in an overlay then the speed of browsing a database, locating a record, or reindexing a file can be greatly affected because the function must be called via the overlay manager. Speed is not usually a problem when overlaying code that has been compiled by Clipper, because the Clipper-code overlay manager is actually part of the Clipper libaries and is very efficient in the manner in which it dynamically overlays pages of P-CODE (Clipper .OBJs). C and ASM native code on the other hand must be overlayed by the linker's overlay manager and can greatly affect performance if an often-used function is not in the "root" memory area. In my opinion, there is no good rule-of-thumb for determining which modules should be overlayed other than the time-tested method of "trial and error". I have spent hours and hours testing different overlay strategies to end up with the just-right balance of speed vs memory optimization. This investment of time has always payed me great dividends because from then on I was confident that I done my homework and that my application was "perfectly tuned". The linkers included in this discussion offer some other fine features that can provide improved speed performance without sacrificing memory performance, however, they require that you utilize some of the advanced features of your computer hardware to get the benefit. 1. OVERLAY CACHING Both RTLINK and BLINKER (release 2.0) support the feature of "caching" overlays to improve runtime speed on computers with EMS or XMS memory. A "cache" is an area of memory that has been pre-allocated for storing code segments that have been overlayed and need to be swapped out to make room in memory for new code. Rather than discarding the code segment and then reloading from disk each time it is needed, it can be reloaded from the "cache" memory. Obviously, memory to memory loading is much faster than disk to memory loading. Blinker -------- Blinker 2.0 uses the following commands for caching with XMS (2.0 and higher): BLINKER CACHE XMS niMax[%], niMinLeave[%] For example the command BLINKER CACHE XMS 50%, 1024 will use a maximum of 50% of all available XMS, but leave at least 1024KB available for other programs. Even faster caching is available by using EMS (extended memory) drivers because memory is handled in large contiguous blocks rather than the smaller 64k "pages" such as the XMS (expanded memory) drivers. Use the following command for caching using EMS (4.0 and higher) BLINKER CACHE EMS niMax[%], niMinLeave[%] Rtlink ------- Rtlink's caching system is only useful when using STATIC overlays. In a previous chapter, I showed you how to overlay the Clipper libraries and third-party libraries using the RELOAD command, BEGINAREA..ENDAREA, and the MODULE command. To improve overlaying performance of C/ASM code in STATIC overlays, use the following Rtlink caching commands: CACHE EXPANDED <memory in K> | <% of available> CACHE EXTENDED <memory in K> | <% of available> 2. OVERLAY POOL SIZE Blinker provides for control of the overlay pool size to allow more code segments to remain in memory and prevent excessive "swapping" in and out of the pool. Increasing the size of the overlay pool will usually have the affect of improving speed performance, but not always. This is another of those "trial and error" options. The disadvantage of increasing the pool size is that the application memory will be reduced, so don't use more overlay pool than is needed for the application. Blinker: BLINKER OVERLAY OPSIZE <nMemoryK> 3. FORCING SEGMENTS TO ROOT Many times when overlaying a library, it is desirable to "exclude" code segments from being overlayed to improve speed. Most often, the reason for the exclusion is because the segment is used globally by many routines or in loops that perform thousands of operations. Sometimes, a segment is just so large that overlaying causes it to be swapped in and out of memory much too often. There are several methods to help you with moving code from an overlayed library to the "root" memory. Rtlink ------- All modules in libraries that are Clipper-compiled .OBJs will be automatically overlayed when the library is declared with the LIB command in your link script. To force a Clipper module from a library to link to the root memory area, use the RESIDENT command and the MODULE command as follows: # MYLIB.lib contains my clipper code LIB MYLIB RESIDENT # force MYFUNCS.obj and MYPROCS.obj to the root MODULE MYFUNCS, MYPROCS # everything from now on is overlayed DYNAMIC Blinker ------- Specified modules in Clipper-compiled libraries and C/ASM libraries which have been overlayed can be forced to root memory by using the MODULE command "outside" the BEGIN... ENDAREA. Example: BEGINAREA # overlay my Clipper application library LIB MYLIB # overlay my C library LIB CFUNCS ENDAREA # force MYFUNCS.obj and MYPROCS.obj to the root MODULE MYFUNCS, MYPROCS # force CFUNCS.obj to the root MODULE CFUNCS Blinker 2.0 also supports a handy feature for automatically forcing segments that are smaller than a specified size to the root area. Very small rountines (less than 32 bytes) take up more root space when they are overlaid, due to the overlay manager table entries, than they would if they were simply left in the root. Additionally, very small overlaid routines execute disproportionately slower, since the time overhead in loading / calling them may be large in comparison with the actual execution time of the routine. BLINKER OVERLAY THRESHOLD <nBytes> will allow you to force all segments that are smaller than <nBytes> into the root. ╔═══════════════════════════════════════════════╗ ║ PROGRAMMING TECHIQUES FOR MEMORY MANAGEMENT ║ ╚═══════════════════════════════════════════════╝ ╔═════════════════════╗ ║ SYMBOL Management ║ ╚═════════════════════╝ For a dynamic-overlay manager to work effectively, it is important that it load overlays at the smallest possible code- segment level, i.e., the procedure/function rather than the entire object. This requires managing the symbol table seperately from the code and data, therefore, these linkers place the symbol table into the root memory area and each function into a separate overlay segment. Symbols cannot be overlayed, therefore it is important that you program in a manner consistent with producing the smallest number of symbols in your Clipper-compiled objects. Here are some tips for reducing the symbol table size in your applications. 1. COMPILE SMALLER SEGMENTS ( C/ASM ) If you are writing code in C or Assembly, then the overlay manager must bring an entire code segment into memory at one time. If you have many large code segments, then the size of the overlay pool may need to be increased. Normally, the requirements of the application dictate the size of the code segments, but it should be noted that a little extra time in optimizing the size of your code will usually pay off if you have a large library of routines. Clipper-compiled code is loaded "dynamically" so this is not a requirement when programming in Clipper. 2. USE CONSTANTS INSTEAD OF MEMVARS Clipper memory variables are always treated as "symbols". Refrain from using a memory variable if a constant is sufficient. For example, an unnecessary symbol can be eliminated by changing the code: escapekey=27 DO WHILE INKEY()#escapekey *clipper code ENDDO to: DO WHILE INKEY()#27 *clipper code ENDDO or: #define K_ESC 27 DO WHILE INKEY()#K_ESC *clipper code ENDDO 3. USE ARRAYS INSTEAD OF MEMVARS Every different Clipper memvar name creates a "symbol", whereas an array name creates only ONE symbol. The following example shows how to save considerable memory in a clipper application by reducing the symbol count with an array. This code produces 10 symbols: mname = name maddress = address mcity = city mstate = state mzip = zip @ 1,1 SAY 'Name ' GET mname @ 2,1 SAY 'Address' GET maddress @ 3,1 SAY 'City ' GET mcity @ 4,1 SAY 'State ' GET mstate @ 5,1 SAY 'Zip ' GET mzip READ REPL name WITH mname, address WITH maddress,; city WITH mcity, state WITH mstate, zip WITH mzip This code produces 6 symbols: PRIVATE gets[5] gets[1] = name gets[2] = address gets[3] = city gets[4] = state gets[5] = zip @ 1,1 SAY 'Name ' GET gets[1] @ 2,1 SAY 'Address' GET gets[2] @ 3,1 SAY 'City ' GET gets[3] @ 4,1 SAY 'State ' GET gets[4] @ 5,1 SAY 'Zip ' GET gets[5] READ REPL name WITH gets[1], address WITH gets[2],; city WITH gets[3], state WITH gets[4], zip WITH gets[5] The following code also produces 6 symbols, however the 5.0 pre-processor is used here to create both DEBUG and NON-DEBUG versions of compiled object code. In the following example the pre-processor is used to convert the 5 memvars to an array when compiling the final version. To compile the below code WITHOUT symbol substitution for debugging purposes, compile as follows: CLIPPER <my app> /dDEBUG To compile the below code WITH symbol substitution for your final application, compile as follows: CLIPPER <my app> The /dDEBUG switch has the same effect as the comand: #define DEBUG in your source code, however, it is much more convenient because it allows you to make debug or non-debug versions of your code simply from the DOS command line or .BAT files. #ifndef DEBUG && compiling non-DEBUG version LOCAL gets[5] #define mname gets[1] #define maddress gets[2] #define mcity gets[3] #define mstate gets[4] #define mzip gets[5] #endif mname = name maddress = address mcity = city mstate = state mzip = zip @ 1,1 SAY 'Name ' GET mname @ 2,1 SAY 'Address' GET maddress @ 3,1 SAY 'City ' GET mcity @ 4,1 SAY 'State ' GET mstate @ 5,1 SAY 'Zip ' GET mzip READ REPL name WITH mname, address WITH maddress,; city WITH mcity, state WITH mstate, zip WITH mzip 4. USE THE SAME NAME MEMVARS WHENEVER POSSIBLE Again, every "different" Clipper memvar in a module creates a symbol. If an object contains several procedures, use the same name for memvars even though they may not perform the same or similar functions. For example, procedure A and procedure B both need 5 memvars. If procedure A declares its memvars with 5 unique names and procedure B declares its memvars with 5 unique names, then 10 symbols are used in the linked application. To eliminate 5 symbols, make sure that procedure B assigns the same name to the memvars as procedure A. This is not possible of course, if the memvars need to be public to both procedures and perform different functions, only if they are private. 5. USE COMPLEX EXPRESSIONS INSTEAD OF MEMVARS The following three lines of codes represents the method that most Clipper programmers choose to accomplish most programming tasks. It makes sense to code this way for readability, but if you are writing a very large application, this technique can be very symbol-intensive. The following three lines of code will read the disk file READ.ME into a memvar named READ_FILE, save the changed code into a file named EDIT_FILE, then write the changed code back to the disk file READ.ME. read_file=MEMOREAD('READ.ME') edit_file=MEMOEDIT(read_file) MEMOWRIT('READ.ME',edit_file) These three lines of code can be replaced by one complex expression which uses no symbols at all. MEMOWRIT("READ.ME",MEMOEDIT(MEMOREAD("READ.ME"))) An additional advantage to coding this way is that less free-pool or VMM memory is used because the process of temporarily storing the text in READ_FILE and EDIT_FILE is completely eliminated. If you find that creating complex expressions such as this are unreadable and hard to maintain then use the 5.0 pre-processor to accomplish the same task as follows: # DEFINE read_file MEMOREAD('READ.ME') # DEFINE edit_file MEMOEDIT(read_file) MEMOWRIT('READ.ME',edit_file) The above code is just as readable as the original three lines of code but will not create any symbols at compile time. Try compiling this code with your Clipper 5.0 compiler and use the /P switch to write the pre-processed code to a .PPO file, then look at the .PPO file to see what is actually compiled. 6. USE LOCALS AND STATICS INSTEAD OF PRIVATES AND PUBLICS Sometimes it is just not possible or practical to write code without using symbols, so if you find yourself in this situation, Clipper 5.0 provides the new feature of "LOCALIZING" symbols to the code segment which is currently being executed rather than placing the symbol in the main symbol table. Local symbols are effectively "overlayed" because they are treated as part of the code segment rather than given a place in the main symbol table. Not only does this save valuable memory but it also improves speed performance because the public symbol table does not need to be searched each time a STATIC or LOCAL symbol is referenced in your code. Of course, if the symbol you are referencing is needed for the entire application or is used in a macro, then it must be declared as PRIVATE or PUBLIC. Symbols which are not declared at all are automatically assumed to be PRIVATE, so make sure you use the LOCAL declaration for all symbols in your code which you do not want to end up in the main symbol table. The following example shows how to save considerable memory in a clipper application by reducing the symbol count with LOCAL declarations. This code produces 6 main memory symbols: PRIVATE mcity,mstate,mzip mcity = city mstate = state mzip = zip @ 3,1 SAY 'City ' GET mcity @ 4,1 SAY 'State ' GET mstate @ 5,1 SAY 'Zip ' GET mzip READ REPL city WITH mcity, state WITH mstate, zip WITH mzip This code produces 3 main memory symbols and 3 local symbols: LOCAL mcity := city, mstate := state, mzip := zip @ 3,1 SAY 'City ' GET mcity @ 4,1 SAY 'State ' GET mstate @ 5,1 SAY 'Zip ' GET mzip READ REPL city WITH mcity, state WITH mstate, zip WITH mzip ╔══════════════════════════════════╗ ║ MEMORY EFFECTS OF DATABASE USAGE ║ ╚══════════════════════════════════╝ Many programmers have taken an affection to "data-driven" programming to speed up the development of custom applications. As a developer of programming tools, rather than custom applications, I find that data-driven programming techniques don't serve me well due to the impact on both speed and memory performance. This seminar is dedicated to memory issues so we won't discuss the speed issues. Since Clipper evolved as an X-Base type language, the common approach to data-driven programming is to create an "engine" or "kernel" .EXE program that retrieves "custom" information about the application from a set of database files. The more sophisticated the system, the more data files are required for configuring the application. It is important to understand the impact of databases on memory usage when designing applications that use many databases. When a database is opened with the USE command, all the field names are placed into the public symbol table. This allows database field names to be used in expressions in the same manner as memvars. Because this symbol table is PUBLIC, the field names will remain in the symbol table, even after the database is closed. Clipper employs no mechanism to remove symbols from the symbol table, only to add them. As each database is opened the symbols are added, thereby reducing the remaining available root memory ( MEMORY(0) ). It is conceivable that an application could run out of memory if many databases are opened and closed. Fortunately, symbols of the same name are "reused" in the symbol table, so if you open and close the same database many times, the symbol memory space will not be reduced. Keeping this in mind, it is a good practice to use databases with fields of the same name. To improve speed operation, some programmers will open data- dictionary databases at the start-up of the program, load the custom program configuration into arrays, then close the databases. This action will reduce the amount of VMM memory needed for file buffers and lower the number of DOS handles required, but the symbols will still be added to the symbol table even though they may never be accessed by the program. A data-driven application in which the databases are used only during the start-up of the application could be re-designed to convert the database information to a text file to generate a "runtime" version of the application that will load the array(s) from a text file rather than databases, thereby eliminating the symbol-table problem.